Dataset expansion: +2.2k targeted markdown rows, 9 new detection labels - #12
Merged
Conversation
Adds --sub-type, --target, --content-filter, --source-label, and --max-skips-multiplier flags to enable targeted streaming from The Stack markdown directory. Used to pull language-specific documentation (markdown documents containing rust/go/etc code blocks) that was underrepresented in the main corpus. Also exposes this via `trainr data pull` click command and adds 49 tests for the new flags, row construction, and content filter regexes. The content filter regex tests include precision tests for the R single-letter regex to verify it does NOT false-match on ruby/rust/rs/ rstudio — the R label is particularly fragile because ```r is easy to confuse with those languages.
Adds csharp, powershell, ruby, php, swift, kotlin, r, lua, graphql to DETECTION_LABELS. These are popular languages that already appeared in the corpus as embedded code blocks but could not be detected by the multi-label head because they had no label. Updates SYSTEM_PROMPT with explicit definitions for each new label, covering positive signals (distinctive syntax/keywords) and anti-signals (common confusions with existing labels like python/ruby, java/kotlin, swift/objc, and especially r vs ruby/rust/rst). Adds "Name alone in prose is not sufficient" caveat to r, swift, and go definitions. These are most susceptible to prose-mention false positives because the words appear in non-code contexts (descriptions, titles). Example JSON template updated to include all 40 labels.
… script Adds ~2,186 markdown documents containing embedded code blocks for 17 languages that were severely underrepresented in the prior corpus. Target-pull strategy uses content-filter regex on The Stack markdown stream (e.g. ```rust, ```go, ```swift) to surface documentation with cross-cutting code examples. Coverage improvements (markdown rows containing each language): - rust: 0 → 340 (largest gap; was literally zero) - go: 9 → 295 - swift: 6 → 173 (new label) - kotlin: 2 → 150 (new label) - lua: 2 → 142 (new label) - r: 1 → 104 (new label) - python: 42 → 192 - objc: 3 → 95 - ruby: 19 → 82 (new label) - graphql: ~0 → 82 (new label) - csharp: 42 → 64 (new label) - php: 17 → 25 (new label) - powershell: 29 → 37 (new label) - (plus incidental boosts to shell/css/ini/dockerfile/sql/html/typescript) Internal deduplication of 12 near-duplicate pairs (templated Docker Hub READMEs, LeetCode writeups, cryptocurrency fork READMEs) via FAISS feature-space L2 threshold 0.1. Existing corpus preserved exactly — NO cross-corpus dedup was performed because the existing golden_train.parquet is not a fixed point under the current dedup pipeline (re-running dedup alone drops 8,053 rows). Filed as arc bug textclf-1dd1.013xtr for future investigation. Adds audit_detection_labels.py — new analysis script that quantifies each detection label's fire rate, cross-fire rate (label != sub_type), label co-occurrence matrix, per-sub_type distribution, and inter-annotator agreement across multiple LLM annotators. Used to validate the dataset expansion via a fresh 5k stratified audit run on three models (gemini-3-flash, sonnet-4.6, gpt-5.4-mini). Post-expansion audit results (golden_train v7): - rust cross-fires: 3 → 29 (+867%) - go cross-fires: 5 → 20 (+300%) - All 9 new labels fire with 100% recall on ground-truth markdown rows - Inter-annotator agreement ≥0.999 for new labels across 3 models - Zero ruby/rust false positives from r regex (critical test) Data tracked via DVC. golden_train.parquet md5 updated from 3cd90fbd60ec30e85fb0a50c29151ac8.dir to c15c01ff8546992d023059923ccb7b30.dir. Old corpus preserved as golden_train.r4.parquet for rollback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR closes a critical dataset gap in the training corpus and adds 9 new detection labels for popular embedded languages. Follow-up to the detection label audit work.
The Gap
The prior corpus had zero markdown rows containing rust code blocks and only 9 containing go. The detection head literally cannot learn to detect rust-in-markdown when no such examples exist. Similar gaps existed for swift (6), kotlin (2), lua (2), and objc (3).
The Fix
Adds
--content-filtersupport topull_real_datato target specific languages when streaming The Stack markdown. Uses this to pull ~2,200 markdown documents containing embedded code blocks for 17 languages.Coverage improvements (markdown rows containing each language):
New Detection Labels (9)
Added to
DETECTION_LABELSwith explicit definitions inSYSTEM_PROMPT:csharp,powershell,ruby,php,swift,kotlin,r,lua,graphqlEach definition includes positive signals (distinctive syntax/keywords) and anti-signals (common confusions with existing labels). The
r,swift, andgodefinitions have aName alone in prose is not sufficientcaveat to prevent prose-mention false positives.Validation
Ran a 3-model 5k stratified audit after the expansion using gemini-3-flash, sonnet-4.6, and gpt-5.4-mini. Results:
Commits
d33416e— Targeted pull infrastructure:--sub-type,--target,--content-filter,--source-label,--max-skips-multiplierflags + click wrapper + 49 tests including R-regex precision tests4cffd46— 9 new detection labels with SYSTEM_PROMPT definitions and caveats7c34e00— Dataset expansion (2,186 new rows) + audit_detection_labels.py analysis script + DVC hash updateData
golden_train.parquetDVC hash:3cd90fbd…→c15c01ff…golden_train.r4.parquetfor rollbackKnown Issue (Filed Separately)
Discovered during this work: the existing
golden_train.parquetis not a fixed point under the current dedup pipeline — re-running dedup on it alone removes 8,053 rows. Worked around by skipping cross-corpus dedup and using only exact-text deduplication for the expansion. Filed as arc bug textclf-1dd1.013xtr for separate investigation.Test plan